Fellow in Quantitative Methodology
London School of Economics
10 minute break
1 hour lunch
10 minute break
Introduction to Large Language Models
Attention Mechanism
from transformers import pipeline
unmasker = pipeline('fill-mask', model='bert-base-uncased')
unmasker("Hello I'm a [MASK] model.")
[{'sequence': "[CLS] hello i'm a fashion model. [SEP]",
'score': 0.1073106899857521,
'token': 4827,
'token_str': 'fashion'},
{'sequence': "[CLS] hello i'm a role model. [SEP]",
'score': 0.08774490654468536,
'token': 2535,
'token_str': 'role'},
{'sequence': "[CLS] hello i'm a new model. [SEP]",
'score': 0.05338378623127937,
'token': 2047,
'token_str': 'new'},
{'sequence': "[CLS] hello i'm a super model. [SEP]",
'score': 0.04667217284440994,
'token': 3565,
'token_str': 'super'},
{'sequence': "[CLS] hello i'm a fine model. [SEP]",
'score': 0.027095865458250046,
'token': 2986,
'token_str': 'fine'}] Important to think about the problem description
Many problems in social science boil down to classification tasks.
In this section, we will focus on classifying news articles into different categories based on their text, but the same principles apply to other classification tasks.
Political Science Applications
Political persuasion and micro-targeting: Hackenburg & Margetts (2023) and Simchon et al. (2024)
Tailoring messages to specific audiences: Mellon et al. (2024) and Velez (n.d.)
Synthetic survey data generation: Bisbee et al. (2023), Sanders et al. (2023) and Simmons & Hare (2023)